One or more aspects of the invention relate generally to determining a quality parameter for a verification environment for a register-transfer level hardware design language description of a hardware design. One or more aspects relate further to a verification system, a method, a computing system, a data processing program, and a computer program product.
New electronic systems, in particular digital chips like a System-on-a-Chip (SoC), are typically designed by using software tools. These allow designing, simulating and testing the functionality before the chip is built as a semiconductor. Typically, a hardware description language or hardware design language (HDL) is used. Producing the actual chip is very expensive. Therefore, intensive testing of the design is required. Typically, simulation-based verification is used for this purpose. A basic idea is verification by redundancy. An alternative design with slight modifications may be used and outputs of the original design and the slightly modified design may be compared. This process may be automated. A checking system, i.e. a checker, for the comparison plays a key role in this process. Challenges lay in stimuli generation for the alternative design, as well as in the checker.
Typically, the basic assumption is made that an optimum of the behavior of the design under test will be captured by the checking system. On the other side, it is also known that checkers are error-prone because they may be wrongly coded, disabled, incomplete and so on.
Simulation-based testing of new hardware designs may be based on mutation testing, which is a fault-based testing technique. Artificial faults may be injected into the design. Then, it is observed whether the checkers may detect the faults. This procedure is based on the general assumption that faults are often caused by programmers of the hardware design. A mutant can be seen as a syntactical change to the original HDL design.
A typical way of hardware mutation testing may be performed in the following way: as input, an HDL design and a test suite for a regression, i.e., a simulation run may be used. For a mutant generation, the HDL design is analyzed and it is determined what kind of mutants are to be inserted into the HDL design. For this, insertion points are determined and typical fault types are used to select or define a mutant to be inserted into the HDL design, which becomes an instrumented design. Then, a fault simulation may be performed by conducting a regression test using the instrumented design. As immediate results, test cases are used, wherein the mutants in signal paths of the HDL design generate a modified behavior of the outputs, which is detected as an error by the checker. On the other side, the mutant may lead to an error propagation that may not be detected by the checking program, i.e., the checker.
A related procedure has been disclosed by the document U.S. Pat. No. 8,010,919 B2, incorporated by reference herein in its entirety. In this document, a method for rating the quality of a computer program whose execution involves an integrated circuit's input and output data being influenced is described. Several steps are performed by the described method. Firstly, a mutated integrated circuit is provided. Then, the mutated integrated circuit's input and output data from the mutated integrated circuit are recorded. A comparison of the mutated integrated circuit and the original integrated circuit is performed. Then, the quality of the computer program on the basis of the results is determined.
However, there are a couple of challenges for mutation testing. One may be the related high simulation costs. This can be demonstrated by a cost estimation of a middle sized unit: assuming 10 fault types, 100 test cases, 1000 insertion points, and a computer system which may be able to run 10,000 cases per day. This may result in a mutation test using 104 mutants, running 106 tests and needing about 100 days. This is totally impractical. Therefore, as mutation based testing often starts at a late stage of verification when a test bed is relatively stable, the number of mutants have to be limited and at the same time, the verification environment needs to produce meaningful results. Therefore, in light of the number of circuits and its complexity, inserting mutants manually into the hardware design is an impractical approach. Additionally, a new compilation may be required for every mutant inserted. And the mutant may be active constantly which may limit the flexibility of the test cases.
Overall, problems may lay in a low efficient insertion point selection (e.g., multiple points may affect the same output path), a re-usage of historical data, and the high volume of data as a result of the verification tests which may not be able to be assessed manually.